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Abstract 

This  thesis  investigated  the  feasibility  of  applying 
artificial  intelligence  technology  within  the  Standard  Base 
Supply  System  (SBSS).  With  seemingly  endless  reductions  in 
manpower  authorizations  within  the  SBSS  and  the  potential  for 
a  continued  loss  of  expert  knowledge,  the  use  of  knowledge- 
based  systems  (KBSs)  was  examined  to  determine  if  these 
systems  could  alleviate  this  loss  of  manpower. 

Literature  related  to  the  fields  of  artificial 
intelligence,  expert  systems,  and  KBSs  was  traced,  yielding  a 
methodology  for  identifying  candidate  problems,  and  for  the 
development,  verification,  and  validation  of  prototype 
KBSs.  This  methodology  was  then  employed  within  the 
research,  culminating  in  the  creation,  verification,  and 
validation  of  a  prototype  KBS. 

The  research  resulted  in  several  conclusions: 

1)  Supply-related  tasks  do  lend  themselves  to  KBS 
development;  2)  A  small-scale  Supply  KBS  Is  feasible  with 
limited  resources;  and  3)  A  small-scale  Supply  KBS  can  be 
validated. 

Several  follow-on  studies  were  recomnended ,  with  the 
thrust  being  that  additional  KBSs  should  be  developed, 
tested,  and  placed  into  operational  use  within  the  SBSS. 
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AN  APPLICATION  OF 
ARTIFICIAL  INTELLIGENCE  TECHNOLOGY 
WITHIN  THE 

STANDARD  BASE  SUPPLY  SYSTEM 


I .  Introduction 


gentry ^  Issue 

The  development  of  artificial  intelligence  and,  more 
recently,  expert  systems  technology  has  led  to  n\imerous 
lucrative  applications  within  the  scientific  and  business 
communities.  It  is  estimated  that  there  are  over  2,000 
operational  expert  systems  in  use  currently  within  the  United 
States,  approximately  80  percent  of  which  support  PC 
applications  (Norville,  1990:24), 

By  providing  the  capacity  to  capture  the  key  decision 
heuristics  of  experts  for  subsequent  use  by  a  wide  range  of 
users,  expert  systems  have  provided  a  valuable  tool  for  the 
management  and  dissemination  of  often-scarce  human  expert 
knowledge.  Expert  systems  have  been  used  in  a  wide  array  of 
applications;  Waterman  summarized  the  "generic  categories  of 
expert  systems  applications"  as  presented  in  Table  1. 

The  tremendous  potential  of  expert  systems  technology 
has  been  exploited  in  nximerous  ways.  Waterman  provides  a 
good  overview  of  several  highly  successful  expert  system 
applications,  as  seen  in  Table  2. 
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Table  1 


EXPERT  SYSTEM  GENERIC  CATEGORIES 


Category 

Problem  Addressed 

Interpretation 

Inferring  situation  descriptions  from 
sensor  data 

Prediction 

Inferring  likely  consequences  of  given 
situations 

Diagnosis 

Inferring  system  malf xinctions  from 
observations 

Design 

Configuring  objects  under  constraints 

Planning 

Designing  actions 

Monitoring 

Comparing  observations  to  expected 
outcomes 

Debugging 

Prescribing  remedies  for  malfunctions 

Repair 

Executing  plans  to  administer 
prescribed  remedies 

Instruction 

Diagnosing,  debugging,  and  repairing 
student  behavior 

Control 

Governing  overall  system  behavior 

(Waterman,  1985:33) 

One  fairly  recent  off-shoot  of  expert  systems  technology 
involves  the  use  of  systems  which  synthesize  the  judgemental 
or  intuitive  knowledge  of  human  experts  with  established 
written  directives  in  an  attempt  to  provide  what  has  been 
referred  to  as  a  knowledge-based  system  (KBS).  (Norville, 
1990:22) . 

While  the  distinction  between  traditional  expert  systems 
and  KBSs  could  be  seen  as  slight,  it  is  nevertheless 
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Table  2 


EXPERT  SYSTEM  SUCCESSES 


Type 

Name 

Application 

Research: 

SYNCHEM2 

Synthesizes  complex  organic 
molecules  without  help  from 
a  chemist 

DENDRAL 

Identifies  molecular  struc¬ 
ture  from  mass  spectral  data 

MACSYMA 

Solves  algebraic  simplifica¬ 
tion  and  integration 
problems 

Business : 

ACE 

Provides  trouble-shooting 
reports  and  analyses  for 
telephone  cable  maintenance 

DELTA 

Helps  diagnose  and  repair 
diesel  electric  locomotives 

SPE 

Diagnoses  inflammatory  con¬ 
ditions  by  interpreting 
scanning  densitometer  data 

XCON 

Configures  VAX  11/780 
computer  systems 

(Waterman,  1985:213) 

important  to  understand  the  difference. 

The  term  expert  system,  in  the  context  of  this  thesis, 
refers  to  a  computer  system  which  has  been  developed  through 
a  complex  interaction  between  a  knowledge  engineei  and  a 
human  expert.  In  this  process,  the  knowledge  engineer 
attempts  to  extract  as  much  of  the  knowledge  and  expertise  of 
the  human  expert  and  to  code  this  information  into  the 
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knowledge  base  of  an  expert  system,  for  subsequent  use  by 
non-experts . 

In  contrast,  the  term  knowledge-based  system  represents 
a  subset  of  expert  system  technology,  with  the  key 
distinction  being  that  in  a  KBS,  the  knowledge  engineer 
extracts  information  from  a  variety  of  sources  (e.g.,  paper 
manuals  and  regulations,  one  or  more  hvunan  experts,  etc.)  to 
shed  light  on  a  particular  subject.  Thus  (as  illustrated  by 
Figure  1),  "virtually  all  expert  systems  are  knowledge-based 
systems,  while  the  converse  is  not  necessarily  true" 
(Waterman,  1985:18). 


Artificial 

Intelligence 

Programs 


Knowledge-Based 

Systems 


< -  Exhibit  intelligent 

behavior  by  skillful 
application  of  heuristics 


Make  domain  knowledge 
explicit  and  separate  from 
the  rest  of  the  system 


Expert 

Systems 


■-  Apply  expert  knowledge 
to  difficult,  real-world 
problems 


(Waterman,  1985:18) 


Figure  1.  Overall  AI  Structure 


KBSs  lend  themselves  well  to  the  rapid  retrieval  and 
display  of  context-specific  information,  such  as  that  found 
in  large  paper  manuals  or  regulations.  Norville  found  that 
"Expert  systems  that  help  staff  wade  through  complex 
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regulations  [as  KBSs  do]  are  a  natural  for  agencies  [which 
possess  a  great  deal  of  paper  guidance]"  (Norville,  1990:22). 

In  addition,  KBSs  which  manage  to  synthesize  the  ability 
to  quickly  access  context-specific  regulatory  guidance  with 
the  ability  to  access  imbedded  human  judgemental  knowledge 
can  provide  a  particularly  potent  combination  for  problem 
solving  in  organizations  characterized  by:  1)  a  relative 
shortage  of  human  experts;  and  2)  a  fairly  rigid,  regulation 
or  manual -driven  set  of  procedures. 

Justification  for  the  Research 

Despite  the  highly  visible  and  demonstratively 
profitable  use  of  expert  system  technology  within  the  science 
and  business  communities  over  the  past  10  years,  the 
application  of  this  technology  has  seen  only  limited  use 
within  the  U.S.  military  logistics  structure.  With  the 
potential  for  more-restrictive  budgets  in  the  future  and  the 
ever-present  drive  within  the  U.S.  military  structure  to  "do 
more  with  less,"  there  exists  a  need  to  incorporate  expert 
system  technology  into  logistics  areas  which  appear  likely  to 
benefit  from  this  technology. 

Recognizing  this  need.  Col  Douglas  Blazer,  currently 
serving  on  the  Air  Staff  at  the  Pentagon  (Office  Symbol  HQ 
USAF/LGX),  has  proposed  via  an  Air  Force  Institute  of 
Technology  Form  53  ("External  Proposal  -  Thesis  Research 
Topic"  -  see  Appendix  A)  that  research  be  conducted  regarding 
the  general  feasibility  of  expert  system  applications  within 
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the  major  logistics  functions  in  the  United  States  Air  Force, 
encompassing  the  Transportation,  Supply,  Maintenance, 
Engineering,  Logistics  Plans  (LOGPLANS),  and  Contracting 
areas . 

In  his  proposal.  Col  Blazer  recommends  that  research  be 
conducted  to  actually  "build  a  prototype  expert  system  for  a 
particular  logistics  process  application"  within  one  of  the 
major  logistics  functions. 

Within  the  United  States  Air  Force,  the  Standard  Base 
Supply  System  (SBSS)  represents  a  logistics  function  which 
exhibits  characteristics  closely  aligned  with  those  described 
above  (in  the  discussion  on  what  type  of  organizations  could 
benefit  most  from  KBS  implementation).  The  SBSS  is  in  itself 
a  complex  system  encompassing  several  different  functional 
organizations;  this  system  is  represented  at  the  base  or  wing 
level  by  the  Supply  Squadron. 

Traditionally,  base-level  Supply  Squadrons  have 
possessed  a  shortage  of  human  experts.  This  shortage  has 
been  exacerbated  in  recent  years  due  to  the  general  drawdown 
in  forces.  According  to  CMSgt  Richard  E.  Grist,  who  is 
responsible  for  managing  the  enlisted  force  within  the  645XX 
(Supply)  career  field  at  the  Air  Force  Military  Personnel 
Center  (AFMPC),  the  problem  is  twofold:  1)  Authorizations 
for  enlisted  Supply  Squadron  members  have  been  cut  steeply  in 
recent  years,  yielding  an  overall  smaller  Supply  force;  and 
2)  The  absolute  number  of  Supply  enlisted  personnel  has  been 
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in  a  state  of  nearly  constant  reduction  over  the  past  two 
years . 

The  net  effect  of  these  two  simultaneous  actions  has 
been  that  Supply  manning  levels,  as  a  percentage  of 
authorisations,  have  remained  relatively  stable  (as  the 
authorisations  have  dropped  at  roughly  the  same  rate  as 
personnel  reductions  have  occurred),  while  the  actual  nvunber 
of  supply  personnel  responsible  for  essentially  the  same 
tasks  has  rapidly  dwindled.  Thus,  while  on  paper  the  Supply 
enlisted  career  field  would  appear  to  be  relatively  healthy, 
it  has  suffered  relatively  deep  cuts  in  recent  years  (Grist, 
1991) . 

Of  particular  note  regarding  the  drawdown  in  the  Supply 
enlisted  force  is  the  fact  that  a  majority  of  the  cuts  have 
come  in  the  form  of  reduced  accessions.  For  example,  in  the 
645X0  Supply  career  field,  accessions  for  1989  were 
approximately  1,600;  accessions  for  this  career  field  in  1992 
are  projected  at  only  500  (Grist,  1991). 

As  a  result  of  this  reduction  in  accessions,  current 
manning  levels  for  3-level  (64530)  Supply  personnel  stand  at 
only  56  percent.  At  the  present  time,  sufficient  numbers  of 
64550  personnel  exist  to  make  up  for  the  shortage  of  the  3- 
level  Supply  personnel.  However,  by  cutting  the  number  of 
accessions  so  steeply,  it  is  felt  that  over  time  the  problems 
associated  with  loss  of  expert  knowledge  will  grow,  as 
experienced  Supply  personnel  separate  or  retire,  and  an 
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inadequate  number  of  less-experienced  personnel  are  available 
to  fill  their  positions  (Grist,  1991). 

Thus,  the  potential  exists  for  an  expertise  "black 
hole”,  in  which  experienced  Supply  technicians  depart  the  Air 
Force,  taking  their  expertise  with  them  without  adequately 
passing  these  skills  on  to  the  relatively  smaller  number  of 
younger  Supply  technicians. 

Despite  this  potential  for  future  problems,  very  few 
inroads  have  been  made  in  the  Supply  discipline  in  the 
development  and  implementation  of  KBS  technology,  which 
represents  one  possible  way  to  capture  the  knowledge  and 
experience  of  departing  Supply  technicians. 

Problem  Statement 

The  current  shortage  of  highly  skilled  base-level  supply 
technicians,  coupled  with  ongoing  and  impending  DOD  manpower 
cuts,  has  the  potential  to  reduce  the  effectiveness  of  base- 
level  supply  organizations.  According  to  CMSgt  John  Babbitt 
(HQ  Air  Force,  LGSS),  Senior  Supply  personnel  at  all  levels 
within  the  Air  Force  must  seek  solutions  to  the  problems 
associated  with  the  ongoing  loss  of  experienced  supply 
technicians  (Babbitt,  1991). 

One  solution  to  this  problem  may  come  with  the 
development  and  implementation  of  KBSs  within  the  supply 
discipline.  Therefore,  this  thesis  will  examine  the 
applicability  and  feasibility  of  implementing  expert  system 
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technology  (in  the  form  of  a  KBS)  to  stem  the  loss  of  expert 
knowledge  within  the  base-level  supply  organization. 

Research  Questions 

To  address  the  research  problem,  the  following  research 
questions  were  posed: 

1.  Do  supply-related  tasks  lend  themselves  to  KBS 
development? 

2.  If  supply-related  tasks  lend  themselves  to  KBSs,  how 
can  a  prototype  KBS  be  developed  within  the  Standard  Base 
Supply  System? 

3.  How  can  this  prototype  KBS  be  validated? 

Research  Objectives 

This  research  was  conducted  with  two  primary  objectives 
in  mind. 

First,  this  research  attempted  to  demonstrate  the 
feasibility  of  developing  a  prototype  KBS  for  a  specific 
logistics  process  application. 

Second,  from  a  more  pragmatic  standpoint,  this  research 
attempted  to  provide  a  concrete  solution  to  a  current  supply- 
related  problem  occurring  within  the  U.S.  Air  Force. 

Scope  of  the  Research 

Since  the  potential  applications  of  KBS  technology 
within  an  organization  as  complex  as  a  base-level  supply 
account  are  virtually  limitless,  it  was  first  necessary  to 
determine  the  scope  of  this  research  or,  more  specifically, 
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the  nvunber  of  KBSs  to  be  developed,  validated  and 
implemented. 

Given  the  limited  time  available  for  this  research  and 
the  inherent  complexity  of  identifying,  designing,  validating 
and  implementing  an  KBS,  this  research  focused  its  efforts 
on  a  single  supply-related  problem. 

To  further  limit  the  scope  of  this  research  to  a 
manageable  level,  a  single  expert  was  selected  to  provide  the 
judgemental  knowledge  to  be  embedded  into  the  KBS.  A  review 
of  past  literature  regarding  the  optimal  number  of  experts  to 
use  in  developing  an  expert  system  (or  KBS)  indicated  that  in 
many  cases  a  single  expert  was  actually  preferred  to  multiple 
experts.  In  this  case,  an  individual  within  Wright-Patterson 
Air  Force  Base's  Supply  Squadron  was  used,  chosen  primarily 
for  the  easy  accessibility  afforded  by  Hright-Patterson  AFB. 
However,  additional  equipment  experts  at  the  base, 

MAJCOM,  and  HQ  USAF  level  were  used  in  the  program-validation 
phase  of  this  thesis. 

Limitations  of  the  Research 

The  restricted  scope  of  this  research,  as  outlined 
above,  necessarily  imposed  several  limitations  on  this  study. 
The  use  of  a  single  expert  in  the  creation  of  this  expert 
system  (while  fairly  common  in  practice),  as  well  as  the 
restriction  to  a  single  base-level  Supply  Squadron,  could 
limit  the  applicability  of  the  finished  product  within  other 
base-level  Supply  Squadrons  in  the  Air  Force.  The  reason  for 
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this  potential  limitation  is  that  Major  Commands  (MAJCOMs) 
are  allowed  a  certain  amount  of  latitude  in  determining 
specific  methods  of  task  performance.  Oftentimes,  specific 
methods  for  performing  supply  tasks  are  MAJCOM  or  even  base- 
specific.  Thus,  the  KBS  which  was  developed  in  an  Air  Force 
Logistics  Command  base- level  Supply  Squadron  may  or  may  not 
be  directly  useable  by  another  Supply  Squadron  within  another 
MAJCOM . 

However,  it  is  believed  that  the  relatively  minor 
deviations  allowed  to  individual  MAJCOMs  may  be  dealt  with 
effectively  within  a  KBS  by  either:  1)  making  the  application 
generic  for  all  Air  Force  supply  accounts  by  not  addressing 
issues  which  are  MAJCOM-specif ic;  or  2)  by  providing  a 
mechanism  for  individual  MAJCOMs  (or  bases)  to  tailor  the 
system  to  their  own  unique  needs.  Further,  this  potential 
limitation  was  partially  eliminated  through  the  use  of  a 
multi-level  validation  process,  in  which  equipment  experts  at 
the  base,  MAJCOM,  and  HQ  USAF  level  were  used  to  validate  the 
KBS. 

Summary 

Chapter  I  provided  a  basic  foundation  of  information  for 
the  research  accomplished  in  this  study. 

It  began  with  a  discussion  of  the  burgeoning  technology 
of  artificial  intelligence  and  expert  systems,  which  have 
provided  valuable  methods  for  the  capture  and  dissemination 
of  often-scarce  expert  knowledge  within  an  organization. 


Next,  it  discussed  the  technology  of  KBSs  in  relation  to 
the  U.S.  Military  logistics  structure,  defining  the  current 
need  to  integrate  this  new  technology  into  the  various 
logistics  functions  in  order  to  increase  efficiency  and  stem 
the  imminent  loss  of  expert  knowledge  within  the  logistics 
community  caused  by  general  reductions  in  manning  levels. 

Narrowing  its  focus.  Chapter  I  next  discussed  one 
specific  U.S.  Military  logistics  function  which  possesses 
characteristics  of  an  organization  which  could  potentially 
benefit  from  the  introduction  of  KBS  technology:  the 
base-level  Supply  Squadron,  represented  by  the  Standard  Base 
Supply  System  (SBSS). 

The  SBSS  represents  a  system  which  encompasses  several 
supply  organizations,  many  of  which  possess  functional  areas 
requiring  the  availability  of  an  expert.  With  expert 
shortages  currently  existing  in  several  supply  functions,  and 
the  strong  possibility  of  future  losses  of  expert  knowledge 
caused  by  reductions  in  manning  levels,  the  SBSS  faces  the 
challenge  of  maintaining  the  necessary  expertise  to  complete 
all  functional  requirements  (Babbitt,  1991). 

One  possible  answer  to  this  problem  comes  in  the 
development  and  use  of  a  prototype  KBS  to  be  used  to  solve  a 
current  supply-related  problem. 

After  this  brief  introduction  into  the  world  of  KBS 
technology  and  discussion  of  the  specific  problem  within  the 
SBSS,  Chapter  I  continued  by  establishing  three  research 
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questions  to  be  answered  in  solving  this  problem.  Further, 
it  described  the  research  objectives,  related  the  overall 
scope  of  the  thesis,  and  gave  several  limitations  anticipated 
in  conducting  this  research. 

Chapter  II  will  provide  a  review  of  current  literature 
regarding  artificial  intelligence  and  expert  systems,  will 
discuss  pertinent  issues  regarding  KBS  development,  will 
examine  selected  methodologies  found  in  the  literature  for 
creating  and  testing  expert  systems  (and  KBSs),  and  will 
provide  a  general  overview  of  the  structure  of  the  BBSS. 
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Overview 

To  conceptualize  the  specific  activities  involved  in  KBS 
development,  it  will  first  be  necessary  to  develop  a 
foundation  of  knowledge  in  the  broader  subject  of  artificial 
intelligence  (AI).  Thus,  Chapter  II  will  begin  with  a 
discussion  of  AI ,  tracing  the  roots  of  this  discipline  from 
inception  up  to  the  point  in  time  at  which  the  subfield  of 
expert  systems  was  created. 

Next,  a  knowledge  of  the  more  specific  area  of  expert 
systems  is  presented,  including  a  brief  outline  of  the 
historical  developments  in  the  field  of  expert  systems. 

Next,  the  focus  of  the  chapter  turns  to  KBSs, 
specifically  discussing  three  individual  topics; 

1)  The  typical  expert  system/KBS  structure; 

2)  Selecting  candidates  for  KBS  implementation;  and 

3)  Management  of  KBSs. 

Finally,  a  brief  overview  of  the  structure  of  the  SBSS 
is  presented,  in  order  to  provide  the  reader  with  a  basic 
understanding  of  the  system  being  discussed. 

ALtlli.cJjl  ?ntel  Ugci>ce 

Artificial  intelligence;  what  is  meant  by  the  term? 
Definitions  of  AI  are  many  and  varied,  as  noted  by  Morris 
Firebaugh  in  his  book  Artificial  Intelligence:  A  Know 1  edge - 
Based  Approach.  Firebaugh  presents  a  good  compilation  of 
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popular  definitions  of  AI ,  each  taken  from  a  different  source 
having  a  slightly  different  point  of  view.  For  example, 
Firebaugh  presents  the  machine-oriented  perspective  of 
Patrick  Henry  Winston,  who  defined  AI  as:  "the  study  of  ideas 
that  enable  computers  to  be  intelligent"  (Firebaugh  1988:12). 

Next,  Firebaugh  quotes  the  ostensibly  mind-oriented 
viewpoint  of  Charniak  and  McDermott  (1985),  who  defined  AI 
as:  "the  study  of  mental  faculties  through  the  use  of 
computational  models"  (Firebaugh  1988:12). 

Peter  Bock's  more  comprehensive  definition  of  AI  (as 
reported  by  Firebaugh)  was:  "the  ability  of  a  human-made 
machine  (an  automation)  to  emulate  or  simulate  human  methods 
for  the  deductive  and  inductive  acquisition  and  application 
of  knowledge  and  reason"  (Firebaugh  1988:12). 

Finally,  Firebaugh  quotes  Professor  Marvin  Minsky  of 
MIT,  who  gave  the  concise  yet  comprehensive  definition  of  AI , 
as:  "the  science  of  making  machines  do  things  that  would 
require  intelligence  if  done  by  men"  (Firebaugh  1988:12). 

h  Brief  Historical  Perspective  M 

The  field  of  artificial  intelligence,  which  continues  to 
receive  a  great  deal  of  attention  and  research  in  today's 
academic  and  business  environment,  had  its  roots  in  the  mid 
1950s  (Allen,  1986:4).  Allen,  paraphrasing  Feigenbaum  and 
Feldman's  classic  AI  book.  Computers  and  Thought .  states 
that : 
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AI  research  began  in  the  mid  1950 's,  spurred 
in  part  by  scientists'  preoccupation  with 
the  now  famous  Turing  Imitation  Game. 

The  Turing  Game  was  a  test  to  determine 
whether  or  not  a  machine  could  fool  a  human 
in  an  interrogation  game.  The  machine  was 
considered  to  possess  intelligence  if  the 
person  performing  the  interrogation  was 
unable  to  distinguish  the  machine's 
responses  from  another  hvunan's. 

(Allen,  1986:4) 

Additionally,  according  to  Schoen  and  Sykes  (1987), 

the  field  of  artificial  intelligence,  or 
at  least  the  name,  was  created  at  a  summer 
conference  in  1956  at  Dartmouth  College  that 
was  organized  by  Marvin  Minsky,  John  McCarthy 
and  Claude  Shannon.  (Schoen  and  Sykes,  1987:2) 

Optimism  for  the  use  of  AI  technology  and  applications 

ran  high  throughout  the  period  of  1950  through  1970,  as 

observed  in  a  Dartmouth  College  study  written  in  1956 

(as  reported  by  Firebaugh,  1988),  which  stated: 

The  study  is  to  proceed  on  the  basis 
of  the  conjecture  that  every  aspect  of 
learning  or  any  other  feature  of  intelligence 
can  in  principle  be  so  precisely  described 
that  a  machine  can  be  made  to  simulate  it. 

(Firebaugh,  1988:18) 

Unfortunately,  this  optimism  would  not  come  to  fruition 
during  the  period  1950  through  1970,  as  evidenced  by  J. 
Lighthill's  statement  in  a  study  funded  by  the  British 
Government  (again,  as  reported  by  Firebaugh,  1988):  "...  in 

no  part  of  the  field  have  the  discoveries  made  so  far 
produced  the  major  impact  that  was  promised"  (Firebaugh, 
1988:18). 

However,  during  this  period,  several  advances  were  made 
which  kept  the  study  of  AI  alive.  For  example,  the  advent  of 
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■ 

list  processing  languages  such  as  FORTRAN,  COMIT,  and  LISP 
created  the  technological  foundation  upon  which  subsequent  AI 
research  was  grounded  (Allen,  1986:5). 

To  summarize  the  shifting  emphasis  within  the  AI 
community  over  the  history  of  this  discipline.  Waterman 
presents  the  following  figure  (Figure  2,  below). 


High 

Use  extensive,  high- 
quality,  specific  know¬ 
ledge  about  some  narrow 
problem  area  to  create 
very  specialized 
programs 

Program 

Power 

Find  general  methods  to 
improve  representation 
and  search  and  use  them 
to  create  specialized 
programs . 

Find  «eneral  meth¬ 
ods  for  probi  eio 
solving  and  use  them 
to  create  general- 
purpose  progiair.^  . 

Low 

_ L 

_ ^ ^ _ 

1950  1970  1980 


(Waterman,  1985:4) 


Figure  2.  Historical  AI  Research  Focus 


The  advancements  in  the  field  of  AI  from  the  1970s  to 
present  include  the  development  of  expert  systems  and, 
subsequently,  KBSs .  Thus,  a  brief  historical  perspective  of 
expert  systems  is  presented  next. 
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Expert  Systems 

Like  AI,  the  term  "expert  system"  has  been  given  a 

number  of  different  definitions  over  the  past  20  years. 

Lindsay  (1988)  defines  expert  systems  as: 

software  programs  that  solve  problems  by 
mimicking  the  ways  in  which  human  beings 
solve  problems.  In  the  strict  terms  used 
by  many  researchers,  expert  systems  apply 
expertise;  they  behave  just  as  human  experts 
do  whose  advanced  training  or  experience 
equips  them  to  do  the  work  of  sevei...,  and 
to  do  it  brilliantly  every  time. 

(Lindsay,  1988:12) 

A  second  definition  of  expert  systems  is  given  by 
Baswas  et  al .  (1988),  who  state  that: 

Expert  or  knowledge  based  [sic]  systems 
are  simply  application  programs  that 
emulate  human  experts  in  specialized 
domains.  Their  ability  to  solve  complex 
problems  is  based  on  the  creation  of  a 
comprehensive  knowledge  base  that  includes 
relevant  facts  and  heuristics  developed  by 
experts  in  a  given  domain.  (Baswas  et  al . , 

1988:235) 

Of  particular  note  in  the  Baswas  et  al .  definition  is 
the  term  heuristics,  which  is  mentioned  frequently  in 
discussions  on  expert  systems.  A  heuristic  is  any  "rule  of 
thumb,  strategy,  trick,  simplification  or  any  other  kind  of 
device  which  drastically  limits  search  for  solutions  in  large 
problem  spaces"  (Feigenbaum  and  Feldman,  1963:6). 


h  Brief  Historical  Perspective  Qt  Expert  Systems 

The  development  of  list  processing  languages  such  as 
FORTRAN,  COMIT,  and  LISP  in  the  1960s  were,  according  to 
Allen,  "crucial  ..o  the  feasibility  of  expert  systems" 
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(Allen,  1986:5).  With  this  computer  language  capability  came 
the  power  to  manipulate  symbols  and  to  come  to  certain 
conclusions  based  upon  these  symbols  (Allen,  1986:5). 

"Work  by  Joshua  Lederberg  in  1964  led  to  the  development 
of  the  DENDRAL  program  at  Stanford  University  under  the 
direction  of  Edward  Feigenbaum”  (Schoen  and  Sykes,  1987:2). 

In  discussing  the  relative  importance  of  the  development  of 
DENDRAL,  Allen  paraphrases  Feigenbaum  and  McCorduck,  stating: 
“DENDRAL  was  a  key  turning  point  towards  other  knowledge- 
based  AI  programs"  (Allen,  1986:5).  Mishkoff  states  that 
Feigenbaum,  by  overseeing  the  development  of  DENDRAL, 
“pioneered  the  rule-based  approach  that  is  currently  popular 
in  the  development  of  expert  systems"  (Mishkoff,  1985:38). 

Speaking  of  the  expansion  of  expert  systems  research, 
Allen  (1986)  states  “Expert  systems  really  began  to  flourish 
in  the  mid  1970 's,  as  AI  research  centers  were  formed  at 
increasing  numbers  of  educational  institutions"  (Allen, 
1986:6).  Allen  goes  on  to  note  that: 

The  decade  of  the  1980s  saw  the  first 
introduction  of  expert  systems  for  daily  use 
in  business  and  industry.  The  first  system 
XCON,  short  for  expert  configurer,  was 
introduced  at  Digital  Equipment  Corporation 
in  1980.  (Allen,  1986:6) 

Summarizing  the  trend  towards  the  continued  use  of 
expert  systems,  Allen  notes  that  “The  trend  of  the  1980s 
towards  increased  use  of  expert  systems  to  address  the  large 
volxune  of  nontechnical  problems  involving  decisions  or 
assessments  is  projected  to  accelerate"  (Allen,  1986:5). 


19 


structure  of  an  Expert  System 

Structurally,  an  expert  system  is  composed  of  three  main 
parts:  a  knowledge  base,  an  inference  engine,  and  an 
explanation  subsystem. 

In  discussing  the  knowledge  base,  Donald  Waterman  notes: 

The  knowledge  in  an  expert  system  is 
organized  in  a  way  that  separates  the  knowledge 
about  the  problem  domain  from  the  system's  other 
knowledge,  such  as  general  knowledge  about 
how  to  solve  problems  or  knowledge  about  how 
to  interact  with  the  user.  This  collection 
of  domain  knowledge  is  called  the  knowledge 
base...  (Waterman,  1985:18) 

The  inference  engine,  a  second  major  component  of  an 
expert  system,  is  quite  simply  the  area  within  the  system 
which  is  responsible  for  searching  through  the  knowledge  base 
in  an  organized  way  (following  set  knowledge  rules)  to  come 
to  a  solution  (Van  Horn,  1986:165). 

Finally,  the  explanation  subsystem  within  an  expert 
system  serves  to  provide  the  user  with  the  ability  to  query 
the  system  to  determine  its  line  of  reasoning  in  coming  to 
its  conclusions  (Keim  and  Jacobs,  1986:9). 

Feigenbaum  and  McCorduck  describe  the  structure  of  an 
expert  system  as  shown  in  Figure  3. 

Essentially,  Figure  3  provides,  in  addition  to  the 
fundamental  structure  of  an  expert  system,  an  overview  of  how 
the  flow  of  knowledge  is  transferred  from  the  source  (human 
expert)  to  the  ultimate  user.  The  knowledge  engineer,  or 
system  designer,  assists  the  process  by  extracting  the 
knowledge  and  heuristic  rules  from  the  human  expert,  creating 
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EXPERT 


USER 


(Feigenbaum  and  McCorduck,  1983:76} 


Figure  3.  Expert  System  Components 


a  knowledge  base  and  inference  system  from  this  information, 
and  overseeing  the  prototype  development  and  validation. 

Allen,  confirming  Feigenbavun  and  McCorduck's  expert 
system  model,  notes  that  the  knowledge  base  contains  both 
basic  facts  and  heuristic  knowledge,  while  the  inference 
engine  "provides  overall  control  of  the  system"  (Allen, 
1986:18). 

O'Leary  (1988)  discusses  the  fiindamental  structure  of  an 
expert  system  as  follows: 

Structurally,  an  expert  system  usually 
consists  of  four  major  components:  a  data 
base,  a  knowledge  base,  an  inference  engine, 
and  a  user  interface.  The  data  base  contains 
the  data  used  by  the  expert  system. . .  The 


21 


knowledge  base  contains  the  knowledge  that 
the  expert  system  uses  to  process  the  data... 
The  inference  engine  is  the  reasoning  approach 
used  in  the  program  to  process  the  knowledge 
base...  The  user  interface  defines  the  manner 
in  which  the  system  elicits  information  and 
explains  its  conclusions.  (O'Leary,  1988:73} 


Candidates  for  Expert  System  Implementation 

According  to  Bielawski  and  Lewand  (1988),  "it  is  quite 
clear  that  certain  kinds  of  tasks  performed  by  experts  lend 
themselves  to  PC-based  expert  systems  technology  better  than 
others"  (Bielawski  and  Lewand,  1988:11).  So  how  does  one 
decide  on  a  specific  application  for  development  of  an  expert 
system? 

Bryant,  writing  on  this  very  subject,  proposes  that  a 
series  of  questions  must  be  asked  before  initiating  an  expert 
system/KBS  project.  These  questions  are  presented  in 
Figure  4. 

Further,  according  to  Bryant: 

An  expert  system,  like  any  software,  is 
desirable  if  it  does  one  of  two  things. 

Either  an  expert  system  should  be  cost 
effective  by  increasing  revenue  or  reducing 
costs  or,  alternatively,  it  should  contribute 
to  the  efficiency  of  an  organization  by 
providing  expertise  that  otherwise  would 
not  be  available.  (Bryant,  1988:32) 

Grimaldi  and  Marcel li  propose  a  slightly  different 

method  for  determining  the  feasibility  of  an  expert  system. 

They  assert  that  determining  feasibility  should  be  the  fifth 

phase  of  a  five-part  plan.  These  parts  are: 

1)  Searching  for  the  expert. 

2)  Characterizing  the  problem. 

3)  Describing  the  knowledge. 
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(Bryant,  1988:33) 


Figure  4.  Criteria  for  Evaluating  Potential  Expert  System 
Applications 

4)  Using  the  knowledge. 

5)  Evaluating  the  feasibility.  (Grimaldi  and 
Marcelli,  1989:27) 

Van  Horn  sets  forth  two  fundamental  principles  to  keep 
in  mind  when  deciding  to  create  an  expert  system.  The  first 
of  these  principles  is,  '*An  expert  system  requires  some  real 
expertise  in  the  field.  Either  you  must  be  the  expert,  or  be 
able  to  build  the  expertise  through  experience"  (Van  Horn, 
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1986:58).  Van  Horn's  second  key  principle  in  developing 
expert  systems  is: 

Expert  systems  rarely  perform  better  than 
the  experts.  The  problems  on  which  expert 
systems  outperform  human  experts  are  not  tasks 
about  which  they  know  more  than  the  humans, 
but  about  which  they  forget  less.  (Van 
Horn,  1986:58) 

Schoen  and  Sykes  highlight  the  need  to  include  criteria 
in  the  planning  process  for  an  expert  system  with  which  to 
determine  whether  or  not  the  expert  system  is  successful 
(Schoen  and  Sykes,  1987:170).  They  state  that: 

Examples  of  criteria  that  might  be 
used  to  define  success  are  that  a  system: 

1.  Solves  representative  problems 
satisfactorily. 

2.  Increases  the  effectiveness  of 
the  user  in  a  demonstrable  way. 

3.  Provides  recognizable  product  enhancement 
or  features. 

4.  Responds  to  a  market  opportunity  or 
pressure  in  a  specific  manner. 

5.  Satisfies  the  check-signer.  (Schoen  and 
Sykes,  1987:170) 

Waterman  discusses  the  limitations  inherent  in  any 
expert  system  project;  Waterman's  discussion  is  portrayed 
in  Figure  5. 

"Just  because  it's  possible  to  develop  an  expert  system 
for  a  particular  task  doesn't  mean  that  it's  desirable  to  do 
so"  (Waterman,  1985:130).  In  order  to  justify  the  time  and 
money  involved  with  undertaking  an  expert  system/KBS  project. 
Waterman  developed  Figure  6. 
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(Nat  eman,  1985 : 181 ) 


Figure  5.  Expert  System  Limitations 


Finally,  Waterman  summarizes  the  necessary  requirements 
for  any  expert  system/KBS  development  project  (see  Figure  7) 


Prgiect 


The  overall  management  of  expert  systems  and  KBSs  is 
generally  considered  to  require  "more  effort  and  insight  than 
comparable,  traditional  information  systems  projects"  (Eliot, 
1989:24).  One  reason  for  this,  according  to  Eliot  (1989), 
results  from  the  inevitable  lag  which  occurs  between  the  time 
that  new  technology  --  in  this  case  KBS  technology  —  becomes 
available  and  the  time  that  managers  of  the  new  technology 
become  proficient  in  its  development  and  overall  maintenance. 
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(Waterman,  1985:130) 


Figure  6.  When  is  an  Expert  System  Justified? 


Norville  contends  that: 

Maintenance  can  be  a  problem  in  these 
[KBS]  systems.  You  can't  go  in  and  change 
things.  You  run  the  risk  of  destroying  the 
logic  in  closed  domains.  Adding  to  or  correcting 
it  [the  KBS]  can  be  difficult.  Maintenance  is 
an  issue  organisations  trill  have  to  address, 
especially  as  knowledge-based  systems  become 
more  complex.  (Norville,  1990:24) 

Martino  recommends  that  the  knowledge  engineer  should 
plan  for  a  built-in  maintenance  capability  in  designing  KBSs, 
including  the  capability  of  "adding  new  rules  as  needed"  and 
"purging  obsolete  rules"  (Martino,  1991:3). 
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Figure  7.  When  is  an  Expert  System  Feasible? 

In  addressing  the  overall  management  of  expert  system 
projects,  Eliot  used  the  case-study  approach  to  describe  the 
successes  and  failures  among  different  methods  of  overseeing 
KBS  projects,  resulting  in  several  useful  points  regarding 
this  issue.  For  example,  Eliot  found  that  "Expert  systems 
should  not  be  used  just  because  the  competition  is  using 
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them"  (Eliot,  1989:25).  He  also  stressed  the  importance  for 
organizations  to  keep  current  in  recent  developments  in 
expert  system  technology,  in  order  to  more  realistically 
judge  the  worth  of  embarking  upon  a  new  KBS  project. 

The  issue  of  KBS  management  becomes  even  more  critical 
within  the  context  of  the  Air  Force  environment,  where 
regulations  and  manuals  are  constantly  updated  and  where 
individual  bases  and  Major  Commands  are  allowed  a  degree  of 
latitude  in  establishing  procedures. 

How  then  could  an  KBS  remain  updat  '  and  useful  within 
the  Air  Force,  and  more  specifically,  the  Air  Force's  base- 
level  supply  account?  In  order  to  answer  this  question,  an 
individual  possessing  experience  in  overseeing  the 
development  and  management  of  a  micro- computer  based  project 
at  the  Major  Command  level  was  contacted.  Mr.  James  Turner, 
currently  serving  on  the  staff  at  HQ  Air  Training  Command  and 
acting  as  the  manager  for  a  command-wide  clothing-issue 
program,  described  the  development  and  management  of  a  micro¬ 
computer  based  program  as  follows: 

While  many  micro-computer  based  programs  are  generated 
by  the  Air  Force  Logistics  Management  Center  at  Gunter  Air 
Force  Base,  programs  may  be  developed  at  any  organizational 
level  by  individuals  with  programming  experience  who  perceive 
an  unmet  computer  requirement.  Once  a  program  is  developed 
and  initially  tested  for  validity  and  accuracy,  the  developer 
may  attempt  to  "sell"  the  program  to  someone  knowledgeable  in 
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the  functional  area  it  addresses,  often  at  the  Major  Command 
level. 

If  a  Major  Command  accepts  the  program  responsibility, 
it  becomes  the  sole  manager/overseer  of  the  program.  An 
individual  knowledgeable  in  both  computer  programning 
techniques  and  supply  procedures  is  usually  assigned  the 
responsibility  of  fielding  questions  and  solving  problems 
discovered  '*in  the  field",  as  well  as  providing  updates  as 
they  become  necessary  to  all  authorized  users  of  the  system. 

A  publication  is  produced  on  a  recurring  basis  which  is 
disseminated  throughout  the  Air  Force,  identifying  the  many 
micro-computer  based  systems  currently  operational  within  the 
Air  Force  and  providing  information  as  to  how  an  organization 
may  be  added  to  the  distribution  list  to  receive  a  desired 
computer  program  and  its  associated  updates.  This 
publication  includes  the  name  and  functional  address  of  the 
individual  or  organization  responsible  for  the  overall 
program  management  for  each  program,  providing  a  vehicle  for 
resolving  program  problems,  receiving  information  regarding 
the  program,  etc. 

Any  tailoring  required  of  a  computer  program  to  bring  it 
in-line  with  specific  Major  Comnand  procedures  is  generally 
completed  by  the  program  manager  (at  Major  Command  level). 
However,  in  the  case  of  a  program  flowing  between  several 
Major  Comnands,  this  tailoring  may  be  done  in  one  of  several 
ways:  1)  By  assigning  a  second  program  manager,  this  one 
serving  within  the  Major  Command  who  would  like  to  make  use 
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of  the  program;  or  2)  By  individuals  at  the  unit  level  who 
wish  to  make  use  of  a  specific  program  developed  within 
another  Major  Command. 

The  simplest  method  of  insuring  cross -command 
applicability  is  to  keep  the  program  at  a  fairly  generic.  Air 
Force  level,  without  incorporating  information  which  is 
allowed  to  be  changed  by  individual  Major  Commands.  If  this 
is  not  possible,  the  complexities  of  distributing  the  program 
across  commands  is  increased  dramatically  (Turner,  1991). 

The  Standard  Base  SudpIv  System  ( BBSS) 

In  order  to  identify  a  specific  functional  area  within 
the  BBSS  in  which  to  apply  KBS  technology,  it  is  necessary  to 
possess  a  fundamental  understanding  of  the  overall  structure 
of  this  system.  The  following  is  an  overview  of  the  SBSS 
structure. 

The  SBSS  represents  a  number  of  different  individual 
organizations  which  together  perform  all  supply-related 
duties  at  the  base  or  wing  level.  The  SBSS  is  lead  by  a 
Chief  of  Supply  (COS),  who  is  generally  a  senior  officer  with 
a  great  deal  of  experience  within  the  64XX  (Supply 
Operations)  career  field. 

The  SBSS  is  broken  down  into  five  separate  branches, 
each  with  a  distinct  role  in  the  mission  of  the  base-level 
supply  account.  These  branches  are:  Management  and  Systems, 
Materiel  Management,  Materiel  Storage  and  Distribution,  Fuels 
Management,  and  Operations  Support  (AFM  67-1,  1987:3-53). 
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The  Management  and  Systems  branch  provides  support 
Ijrimarily  to  the  Chief  of  Supply,  and  is  comprised  of  the 
following  sections:  Customer  Service  and  Training; 
Procedures  and  Analysis;  Funds  Management;  Computer 
Operations;  Ii: 'entor^  ,  Document  Control;  and  Administration 
(AFM  67-1,  1987:2-53). 

The  Materiel  Management  Branch  retains  overall 
responsibility  for  all  requirements  identification, 
requisitioning,  equipment,  and  inventory  stockage  issues 
within  the  SBSS.  It  is  comprised  of  the  following  sections: 
Equipment  Management;  Stock  Control;  and  Retail  Sales  (AFM 
67-1,  1987:2-67). 

The  Materiel  Storage  and  Distribution  branch  maintains 
tite  responsibility  for  the  efficient  and  effective  handling 
and  storage  of  all  incoming  and  outgoing  property.  The 
branch  is  made  of  the  following  sections;  Inspection; 
Receiving;  Storage  and  Issue;  Pickup  and  Delivery;  and  Bench 
Stock  Support  (AFM  67-1,  1987:2-83). 

The  Fuels  Management  Branch  oversees  the  management  of 
all  petroleums,  oils  and  lubricants  (POL)  used  by  wing  or 
base  organizations.  It  is  made  up  of  the  following  sections 
Fuels  Mobility  and  Training;  Quality  Control  and  Inspection; 
Fuels  Operations;  Cryogenics;  and  Accounting  and 
Administration  (AFM  67-1,  1987:2-87). 

Finally,  the  Operations  Support  Branch  is  responsible 
for  direct  customer  support,  maintaining  the  following 
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sections:  Demand  Processing;  Mission  Support;  and  Repair 
Cycle  Support  (AFM  67-1,  1987:2-75). 

Figure  8  illustrates  the  overall  SBSS  organization. 


(AFM  67-1,  1987:Chap  2) 


Figure  8.  Overall  Structure  of  the  SBSS 


Sanmary 

Chapter  II  defined  and  outlined  the  short  history  of 
artificial  intelligence  and  expert  systems  by  tracing  the 
literature  on  these  topics.  It  also  provided  insight  into 
the  inner  workings  of  expert  systems/KBSs  by  examining 
literature  regarding:  1)  the  structure  of  these  systems; 
and  2)  how  to  identify  potential  candidates  for  expert  system 
implementation . 

Chapter  II  further  discussed  the  problems  noted  in  past 
studies  related  to  the  maintenance  and  management  of  KBSs, 
focusing  specifically  on  maintenance  of  systems  within  the 
USAF. 
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shifting  directions.  Chapter  II  also  provided  an 
overview  of  the  SBSS  in  an  effort  to  impart  a  fiindamental 
understanding  of  this  complex  system  to  the  reader;  this 
knowledge  was  critical  during  the  problem  selection  phase  of 
the  research  discussed  in  Chapter  IV. 

Chapter  III  will  present  an  overview  of  the  methodology 
used  in  this  thesis  to  seek  answers  to  the  original  research 
questions  posed  in  Chapter  I. 
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Ill .  Methodology 


Overview 

Chapter  III  provides  a  discussion  of  the  methodology 
selected  for  use  in  this  thesis.  It  traces  literature 
related  to  various  methodologies  which  have  been  used  in  past 
research  studies  in  developing  and  validating  expert 
system/KBS  projects. 

Chapter  III  culminates  with  the  presentation  of  a  table 
which  summarizes  the  specific  methodology  used  in  this 
thesis . 

Introduction 

While  the  technology  of  expert  systems  has  been  widely 
available  for  a  relatively  short  time,  numerous  expert 
systems  have  been  conceived,  developed,  tested,  and 
implemented.  As  a  base  of  experience  developed  in  the  best 
methods  to  conduct  the  process  of  creating  an  expert 
system/KBS,  a  pattern  began  to  emerge.  Essentially,  all 
developmental  aspects  of  a  KBS  can  be  observed  to  follow  a 
set  path.  Different  authors  have  given  slightly  different 
names  to  some  of  the  developmental  stages  in  expert 
system/KBS  evolution;  however,  reviewing  the  literature 
reveals  that  the  overall  structure  involved  with  creating  an 
expert  system  remains  consistent  across  virtually  all 
applications . 
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Pev^lQPiRq  An  Expert  System/ KBS 


Before  an  expert  system/KBS  project  can  begin,  a 
suitable  problem  must  be  discovered  on  which  to  apply  the 
computer  techniques.  Thus,  the  first  step  in  KBS  development 
is  identification  of  a  good  candidate  problem.  Methods  for 
screening  potential  KBS  applications  have  been  developed  by 
nvimerous  researchers;  these  methods  were  discussed  in  Chapter 
ZI.  Once  the  candidate  KBS  is  selected,  the  development 
process  may  begin.  Providing  a  comprehensive  overview  of  the 
KBS  development  process,  Allen  (using  information  presented 
by  Waterman)  discusses  the  development  process  as  presented 
in  Table  3,  below. 

While  Table  3  provides  a  good  macro  view  of  the  KBS 
development  process,  a  more  complete  step-by-step  outline  of 
the  phases  involved  with  developing  an  expert  system/KBS  are 
presented  by  Freiling  £t  al . . 

According  to  Freiling  £t  al . .  an  expert  system/KBS 
project  may  be  broken  down  into  two  phases:  the  "knowledge 
definition  phase"  and  the  "prototype  implementation  phase" 
(Freiling  et  al . ,  1985:155-157). 

Further,  the  knowledge  definition  phase  can  be  broken 
down  into  three  distinct  steps,  as  follows: 

Step  1:  Familiarization.  During  this  step,  the  person 
acting  as  the  knowledge  engineer  attempts  to  determine  the 
scope  of  the  project  and  to  understand  the  fundamental  nature 
of  the  problem  to  be  solved.  Several  data  collection  methods 
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Table  3 


KBS  DEVELOPMENT  OVERVIEW 


Method 

Description 

On-site  observation 

Watch  the  expert  solving  real 
problems  on  the  job. 

Problem  discussion 

Explore  the  kinds  of  data,  knowledge, 
and  procedures  needed  to  solve 
specific  problems. 

Problem  description 

Have  the  expert  describe  a  proto¬ 
typical  problem  for  each  category 
of  answer  in  the  domain. 

Problem  analysis 

Present  the  expert  with  a  series  of 
realistic  problems  to  solve  aloud, 
probing  for  the  rationale  behind 
the  reasoning  steps. 

System  refinement 

Have  the  expert  give  you  a  series  of 
problems  to  solve  using  the  rules 
acquired  from  the  interviews. 

System  examination 

Have  the  expert  examine  and  critique 
the  prototype  system’s  rules  and 
control  structure. 

System  validation 

Present  the  cases  solved  by  the 
expert  and  prototype  system  to 
other  outside  experts. 

(Allen,  1986:79) 

may  be  used  during  this  step,  including  informal  interviews 
with  the  expert  selected,  simple  observation  of  the  expert  at 
work,  and  initial  identification  of  other  sources  of  domain 
information  (e.g.,  regulations,  manuals,  etc.)*  This  step  is 
deemed  complete  when  the  knowledge  engineer  is  able  to 
combine  all  knowledge  rules  gained  into  an  initial  knowledge 
base  (Freiling  et.  al .  .  1985:155-156). 
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step  2:  Organizing  knowledge.  This  step  picks  up  where 
step  1  left  off,  with  the  knowledge  engineer  examining  the 
initial  knowledge  base  created  for  patterns  or  regularities. 
The  knowledge  engineer  then  attempts  to  formulate  the 
regularities  fovmd  into  the  first  of  the  heuristic  decision 
rules,  stated  on  paper  in  simple  if-then  form  (Freiling  et 
al . .  1985:156-157). 

Step  3:  Representing  knowledge.  Once  step  2  is 
complete,  and  all  initial  heuristic  decision  rules  have  been 
formulated  on  paper  (and  verified  with  the  human  expert),  the 
knowledge  engineer  decides  how  the  knowledge  can  be 
represented  within  the  expert  system.  (Freiling  al . . 
1985:157) . 

Once  steps  one  through  three  are  complete,  the  second 
phase  of  the  project,  referred  to  as  the  Prototype 
Implementation  Phase,  can  begin.  According  to  Freiling  et 
Hi » i 

Once  the  external  and  internal  knowledge 
base  formats  have  been  defined,  they  can  be 
used  to  guide  the  implementation  of  a  prototype 
expert  system.  The  implementation  process 
consists  of  acquiring  the  knowledge  base, 
building  an  inference  engine,  and  building  an 
appropriate  interface.  (Freiling  et  al . . 

1985:157-158) 

Thus,  the  prototype  implementation  phase  also  involves 
three  steps. 

Step  1:  Acquiring  knowledge.  During  this  step  of 
prototype  implementation,  specific  knowledge  is  collected  and 
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processed  into  a  data-base  structure,  which  serves  as  the 
knowledge  base  for  the  expert  system. 

Step  2:  Inference  strategy  design.  In  this  step,  the 
knowledge  engineer  choses  a  specific  inference  engine  with 
which  to  drive  the  entire  expert  system,  and  completes  the 
actual  coding  of  the  software  which  forms  the  core  of  the 
expert  system. 

Step  3:  Interface  design.  Freiling  et  al .  describe 
this  step  by  saying,  "Generally,  this  [step]  involves  trying 
to  discover  what  parts  of  the  task  are  routine  and  can  be 
handled  in  an  effective  interface"  (Freiling  et  al . . 
1985:158).  Thus,  in  this  final  step,  the  system  is  tailored 
to  meet  the  needs  of  the  ultimate  users. 

V^lida.  si  the  Expert  System 

Once  the  prototype  expert  system  is  developed,  it  should 
then  go  through  a  two-part  validation/verification  process  to 
ensure  it  is  functioning  as  intended. 

The  two  processes,  validation  and  verification,  might 
seem  to  be  one  and  the  same  process.  However,  in  this 
instance,  they  are  two  distinct  processes.  O'Keefe  £t  al . 
discusses  the  fundamental  differences  between  the  validation 
and  verification  processes  as  follows: 

Simply  stated,  validation  refers  to 
building  the  right  system  (that  is, 
substantiating  that  a  system  performs 
with  an  acceptable  level  of  accuracy), 
whereas  verification  refers  to  building 
the  system  "right"  (that  is,  substantiating 
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that  a  system  correctly  implements  its 
specifications).  (O'Keefe  ^t  al . .  1987:82) 

Thus,  the  expert  system/KBS  being  developed  should  be 

initially  validated  to  ensure  the  program  has  been 

successfully  "de-bugged"  and  thus  returns  reliable 

(internally  consistent)  answers  to  test  problems.  This 

validation  can  be  conducted  in  several  different  ways,  with 

perhaps  the  most  common  method  being  a  rigorous  test  of  the 

computer  coding  using  problems  where  the  answer  is  known  in 

advance. 

A  second  test  of  the  system/KBS 's  usefulness  comes  in 
the  form  of  the  verification  process,  during  which  the 
question  is  asked  "have  we  built  the  right  model?" 

Oftentimes  this  question  is  very  difficult  to  answer 
satisfactorily,  due  to  the  very  nature  of  the  problem  under 
consideration.  Allen  notes  that  "The  very  nature  of  expert 
systems  implies  that  there  is  no  clear  cut  right  or  wrong 
answer  to  which  the  system  addresses"  (Allen,  1986:83).  On 
this  topic.  Waterman  notes: 

Evaluating  an  expert  system  is  difficult 
because  there  may  be  no  formal  way  to  prove  a 
given  answer  is  correct  or  the  best  possible 
choice.  The  validity  of  the  answer  may  depend 
on  the  persuasiveness  of  the  argument  given  to 
support  that  answer.  (Waterman,  1985:198) 

However,  despite  this  ambiguity  in  determining  the 

usefulness  of  a  computer  model  via  the  verification  process, 

an  attempt  must  be  made  to  demonstrate  that  the  right  system 

has  been  built  to  solve  a  particular  problem. 
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Waterman  presents  an  answer  to  this  dilemma  in  his 
discussion  of  one  method  for  completing  the  verification 
process;  Waterman  states: 

The  knowledge  engineer  presents  the... 
prototype  system  to  other  experts.  This  provides 
a  way  to  compare  strategies  of  different  experts 
and  find  essential  points  of  disagreement. 

(Waterman,  1985:160-161) 

Summary  of  Methodology  Used  in  the  Thesis 

Compiling  all  relevant  methodological  considerations 
presented  in  the  literature.  Table  4  presents  an  overview  of 
the  specific  methodology  which  was  used  to  answer  the 
research  questions  posed  in  Chapter  I. 

chapter  III  provided  a  comprehensive  overview  of  the 
steps  taken  in  this  thesis  to  discover  answers  to  the 
research  questions  posed  in  Chapter  I.  It  briefly  traced 
past  methodologies  which  have  been  used  in  KBS  development, 
and  concluded  with  a  summary  of  the  specific  methodology  used 
in  this  research. 

The  information  presented  in  Chapter  III  sets  the  stage 
for  the  more-specific  discussion  which  takes  place  in  Chapter 
IV.  Chapter  IV  discusses  the  implementation  of  the 
methodology  established  in  Chapter  III,  and  provides  findings 
to  the  research  questions  posed  in  Chapter  I. 
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Table  4 


OVERVIEW  OF  SPECIFIC  METHODOLOGY  USED 


Research  Process 


Specific  Methodology 


Step  1: 

KBS  Candidate  Selection 


-  Research  historical  problem 
areas  within  Base  Supply. 

-  Select  the  one  best 
candidate  for  KBS 
development . 

-  Discuss  expert  system/KBS 
implementation  with  an 
expert  in  the  problem  area 
identified,  to  insure 
need/f easibi lity . 


Step  2; 

Problem  Familiarization 


-  Discuss  problem  with  the 
expert . 

-  Collect  all  knowledge  avail¬ 
able  on  the  problem  (from 
the  expert,  manuals,  regu¬ 
lations,  etc.). 


Step  3: 

Prototype  Development 

-  De-bug  program  -  eliminate 
syntactical  problems. 


-  Select  expert  system  shell 
to  be  used  in  program. 

-  Create  the  initial  knowledge 
base  from  information 
gathered  in  step  2. 


Step  4:  -  Present  completed  proto¬ 

type  to  other  experts  to 

Validation  and  Verification  validate  assumptions/ 

usefulness  of  the  program. 

-  Test  the  program  on  novices 
in  the  field 
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IV .  Findings  and  Discussion 


Overview 

Chapter  IV  provides  an  overview  of  the  specific  actions 
taken  by  the  researcher  in  answering  the  original  research 
questions  posed  in  Chapter  I.  As  developed  in  Chapter  III,  a 
four*step  methodology  was  employed,  briefly  summarized  as: 
Step  1:  KBS  Candidate  Selection 
Step  2:  Problem  Familiarization 
Step  3:  Prototype  Development 
Step  4:  Validation  and  Verification 

These  processes,  which  eventually  led  to  the  creation 
and  validation  of  the  prototype  KBS  named  the  "Equipment 
Management  System  Management  Advisory  Program"  (EMS  MAP),  are 
presented  below. 

Step  1:  KBS  Candidate  Selection 

The  first  issue  dealt  with  the  selection  of  an  existing 
supply-related  problem  in  which  to  apply  a  KBS. 

An  answer  to  this  problem  came  in  the  form  of  a  draft 
report  prepared  by  the  Air  Force  Logistics  Management  Support 
System  (LIMSS)  for  the  Air  Force  Logistics  and  Engineering 
Information  Systems  Division.  This  report  provided  a  summary 
of  a  survey  undertaken  to  identify  existing  KBS  applications 
and  to  discuss  "the  feasibility  and  benefits  of  specific 
candidate  logistics  (KBS)  applications"  (U.S.  DOT,  1990:iii). 
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Specifically,  the  report  issued  by  the  LIMSS  was 
initiated: 

in  an  effort  to  identify  AI  applications 
with  a  significant  payback...  A  number  of 
candidate  applications  have  been  identified 
through  interviews  with  potential  users, 
government  organizations  responsible  for 
developing  [KBS]  applications  and  represent¬ 
atives  of  the  LIMSS  program.  (U.S.  DOT,  1990:iii). 

The  report  investigated  potentially  lucrative  KBS 

applications  within  the  following  distinct  fimctional  areas: 

Supply;  Munitions;  Maintenance;  Transportation;  Engineering 

and  Services;  and  Command  and  Control. 

In  the  Supply  functional  area,  four  high-priority  KBS 

candidates  were  identified,  selected  for  their  high  potential 

for  improved  efficiency  via  manpower  savings.  Of  these  four 

recommended  KBS  applications,  the  one  which  appeared  to  offer 

the  most  appropriate  problem  (in  terms  of  degree  of 

difficulty  and  familiarity  of  the  author  to  the  general  topic 

considered)  was  one  proposed  by  Lt  Col  Peterson  from  the  Air 

Force  Logistics  Management  Center  (office  symbol  AFLMC/LGK) . 

Lt  Col  Peterson's  proposal  was  directed  at  improving  the 

ability  of  base-level  equipment  custodians  in  completing 

their  day-to-day  custodian  duties.  These  duties  generally 

involve  interfacing  with  the  Equipment  Management  Section 

(EMS)  within  the  base-level  supply  account  in  ordering, 

maintaining,  and  overseeing  their  organization’s  equipment 

account . 
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In  his  proposal,  Lt  Col  Peterson  states: 

The  Logistics  Management  Center  has 
identified  a  need  for  a  system  to  assist 
in  ordering  items  of  equipment.  There  are 
a  large  number  of  regulations  governing  how 
to  place  an  order,  and  mistakes  are  often 
made  [by  equipment  custodians]  in  preparing 
the  required  forms.  This  results  in  delays 
as  incorrect  order  forms  are  returned  for 
resubmission.  The  situation  is  further 
affected  by  the  high  turnover  of  staff. 

Further  investigation  into  the  problem  described  by  Lt 

Col  Peterson  tangentially  led  to  a  report  prepared  by  Captain 

Jeff  Bailey  from  the  Air  Force  Logistics  Management  Center 

(AFLMC)  in  November  1989  titled  ’’Analysis  of  Base-Level 

Supply  Customer  Training".  Lt  Col  Peterson  served  as  a  team 

member  in  the  preparation  of  this  report. 

Captain  Bailey's  report  was  the  result  of  a  HQ  USAF/LEY 

tasking  to  the  AFLMC  to  review  the  Air  Force  supply  customer 

training  program.  Captain  Bailey’s  stated  objective  for  the 

study  was  ’’to  analyze  the  supply  customer  training  program, 

determine  customer  training  needs,  and  recommend  improvements 

to  the  program  to  meet  those  needs"  (Bailey,  1989:i).  In  his 

report.  Captain  Bailey  notes  that  ’’a  common  perception  among 

supply  customers  and  trainers  is  that  supply  customer 

training  is  inconsistent  from  one  base  to  another"  (Bailey, 

1989:1) . 

This  inconsistency  in  training  was  resulting  in 
inconsistent  application  by  supply  users  (e.g.,  equipment 
custodians)  in  the  performance  of  supply  duties.  In  a  broad 
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sense,  the  knowledge  required  of  supply  customers  (including 
equipment  custodians)  includes: 

1)  a  basic  understanding  of  computer-generated  listings 
received  from  base-level  supply; 

2)  an  ability  to  correctly  fill  out  required  supply 
paperwork; 

3)  an  ability  to  keep  their  supply  records  in  order; 

4)  a  knowledge  of  who  they  may  contact  within  the 
base-level  supply  accoxint  to  answer  their  questions 
(Bailey,  1989:1). 

Captain  Bailey  determined  (and  listed  in  Appendix  D  to 
his  report)  what  he  considered  to  be  the  most  important 
topics  to  be  covered  during  the  initial  equipment  custodian 
training  session.  These  topics  were: 

1.  Equipment  Management  Section  (EMS)  points  of 
contact . 

2.  The  Air  Force  Equipment  Management  System  (AFEMS). 

3.  Equipment  responsibilities. 

4.  Allowances  and  authorizations. 

5.  How  to  read  a  Table  of  Allowances  (TA) . 

6.  Equipment  turn-in  procedures. 

7.  How  to  fill  out  an  AF  Form  2005. 

8.  How  to  fill  out  an  AF  Form  601. 

9.  Custodian  Authorization/Custody  Receipt  Listing 
(CA/CRL) . 

10.  How  to  establish  and  maintain  an  equipment 
custodian  file  (Bailey,  1989:15). 
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According  to  Captain  Bailey,  many  of  the  individual 
supply  customers  who  were  surveyed  as  a  part  of  his  study 
felt  that  the  best  way  to  provide  the  required  training  and 
information  was  through  the  use  of  a  short  classroom  lecture 
combined  with  a  source  of  information  (such  as  a  hand-out) 
which  the  customer  could  take  back  to  his  or  her  workplace 
which  would  act  as  a  guide  for  future  supply  actions. 

From  the  above  information,  it  appeared  likely  that  a 
computerized  product  in  the  form  of  a  KBS  which  would 
contain  a  compilation  of  information  (both  from  an  expert  and 
from  manual/regulation  guidance)  could  improve  the  ability  of 
equipment  custodians  to  complete  the'r  day-to-day  supply 
tasks.  To  verify  the  applicability  of  this  type  of  a 
project,  and  to  determine  the  feasibility  of  producing  such  a 
product,  an  individual  knowledgeable  in  the  training  of 
equipment  custodians  was  consulted. 

Ms.  Jody  Taylor  (GS-11),  the  head  of  the  Wright- 
Patterson  AFB  Base  Supply  Training  Section,  indicated  that 
the  premier  expert  on  training  issues  regarding  equipment 
management  was  SSgt  Jackson,  who  also  worked  in  the  Training 
Section.  SSgt  Jackson  was  contacted  and  an  initial  interview 
was  held,  yielding  the  following  information: 

1)  Approximately  900  equipment  accounts  were  currently 
operational  at  Hright-Patterson  AFB. 

2)  SSgt  Jackson  was  responsible  for  conducting  all 
training  sessions  for  newly  assigned  equipment  custodians. 
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SSgt  Jackson  possessed  a  great  deal  of  experience  in 
equipment  matters,  and  was  generally  regarded  as  the  expert 
in  this  area  by  equipment  custodians  at  Hright-Patterson  AFB. 

3)  SSgt  Jackson  had  developed  a  handbook  to  be  given  to 
newly  trained  equipment  custodians  to  serve  as  a  basic  guide 
for  future  equipment  actions.  However,  SSgt  Jackson  reported 
that  the  use  of  the  handout  was  limited,  and  that  he 
routinely  received  phone  calls  from  equipment  custodians 
asking  for  further  assistance  on  specific  equipment  matters. 

4)  The  information  which  would  serve  as  the  knowledge 
base  for  this  program  was  readily  available  in  several 
different  Air  Force  Regulations  and  Manuals.  Also,  SSgt 
Jackson  was  willing  to  serve  as  the  expert  in  this  project. 

From  the  above  information,  it  was  determined  that  the 
KBS  project  under  consideration  was  both  feasible  and 
desirable.  Thus,  this  project  was  determined  to  be  the  one 
which  would  be  used  in  conducting  the  research  study. 

Step  2:  Problem  Familiarization 

The  problem  familiarization  step  took  place  over  the 
course  of  approximately  three  months,  and  involved  gathering 
information  from  several  sources. 

The  first  source  of  information  was  SSgt  Jackson,  the 
individual  deemed  to  be  the  expert  in  equipment  management 
matters  at  Hright-Patterson  AFB.  Through  interviews  with 
SSgt  Jackson,  the  "big  picture"  of  the  equipment  management 
function  gradually  emerged,  and  a  foundation  of  understanding 
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regarding  the  tasks  required  of  an  equipment  custodian  was 
developed. 

SSgt  Jackson  also  provided  the  researcher  with  all 
pertinent  references  for  equipment  information  to  be  found  in 
Air  Force  Regulations  and  Manuals,  and  also  provided  the 
researcher  with  a  copy  of  the  locally  developed  equipment 
handbook  given  to  newly  trained  equipment  custodians.  These 
additional  sources  were  consulted  over  the  entire  knowledge 
familiarization  phase,  and  provided  useful  information  to  be 
incorporated  into  the  prototype  KBS. 

Captain  Bailey's  research  project  was  discussed  with 
SSgt  Jackson,  who  concurred  that  the  10  items  identified  by 
Captain  Bailey  as  vital  training  topics.  Further,  SSgt 
Jackson  recommended  that  the  equipment  custodian  KBS  to  be 
developed  should  attempt  to  follow  these  general  topics 
in  presenting  its  information. 

Step  3;  Prototype  Development 

The  actual  prototype  development  step  commenced 
once  the  application  to  be  completed  was  determined  and  all 
knowledge  available  on  this  topic  was  gathered. 

To  begin  this  step,  a  determination  was  required  as  to 
which  of  the  several  commercially  available  expert  system 
shells  would  be  used  to  actually  code  the  program.  The  PC- 
based  expert  system  shell  "VP-Expert"  was  chosen,  primarily 
due  to  its  ease  of  operation  and  the  researcher's 
familiarization  with  this  shell.  Additionally,  built-in 
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features  providing  for  such  things  as  windows,  colors,  and 
mathematical  operations  made  "VP-Expert"  a  good  choice  for 
this  project. 

Next,  the  actual  coding  of  the  knowledge  base  began. 

It  was  decided  to  create  an  overall  structure  which  was  in¬ 
line  with  that  proposed  by  Captain  Bailey  for  completing 
the  equipment  custodian  training  session.  Thus,  major  topics 
available  from  the  main  menu  were: 

1.  "Welcome  New  Users."  This  portion  of  the  EMS  MAP 
provided  basic  program  information  to  new  users,  including 
the  purpose  of  the  program  and  fundamental  concepts  related 
to  its  use. 

2.  "AFEMS  Overview."  This  module  provided  basic 
information  regarding  the  Air  Force  Equipment  Management 
System  (AFEMS). 

3.  "Points  of  Contact."  This  module  gave  the  user  the 
names  and  duty  phone  numbers  of  key  personnel  within  Base 
Supply,  including  individuals  within  the  Equipment  Management 
Section  and  the  Customer  Service  Section. 

4.  "Definitions."  This  portion  enabled  the  user  to 
select  from  a  menu  of  equipment- related  terms,  providing  a 
short  definition  of  the  term  to  assist  the  equipment 
custodian. 

5.  "Acronyms."  Similar  to  the  "Definitions"  module, 
this  portion  provided  a  menu  of  supply  acronyms  for  the  user 
to  select  from;  upon  selection,  the  program  displayed  a  brief 
explanation  of  the  acronym  selected. 
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6.  "Responsibilities."  This  module  provided  the  user 
with  an  overview  of  the  responsibilities  of  the  equipment 
custodian,  the  organizational  commander,  the  responsibility 
center  manager,  and  the  cost  center  manager. 

7.  "Folder  Maintenance."  This  portion  of  the  EMS  MAP 
provided  a  step*by-step  set  of  procedures  for  maintaining  the 
required  equipment  custodian  folder. 

8.  "CA/CRL  Overview."  This  module  provided  the  user 
with  an  overview  of  the  Custodian  Authorization/Custodial 
Receipt  Listing  (CA/CRL).  This  listing  is  used  extensively 
by  equipment  custodians;  an  understanding  of  its  use  is 
essential  in  maintaining  an  equipment  account. 

9.  "Help  with  Forms."  This  portion  of  the  program 
provided  the  user  '-h  basic  information  on  the  various  forms 
required  of  an  equipment  custodian.  It  also  provided  advice 
to  the  custodian  on  which  form  he/she  should  use,  after 
asking  the  user  several  questions  to  determine  the 
appropriate  form. 

This  menu  system  provided  the  overall  structure  for  the 
prototype  KBS;  each  menu  item  served  as  a  separate  module  for 
the  knowledge  engineer  (the  researcher,  in  this  case)  to  code 
into  the  program. 

The  vast  majority  of  the  information  coded  into  the 
RBS's  knowledge  base  came  directly  from  written  materials; 

Air  Force  Manual  67-1,  Air  Force  Regulation  67-23,  and  SSgt 
Jackson's  locally  developed  equipment  handbook  were  each  used 


50 


extensively  in  this  phase  of  program  development.  The  expert 
•knowledge  of  SSgt  Jackson  was  used  primarily  as  an  overall 
"control  structure".  This  knowledge  provided  coherence  and 
flow  to  the  program  from  a  h\iman  point  of  view,  ensuring  the 
information  was  being  presented  in  a  useful,  easily 
accessible  fashion,  from  an  equipment  manager's  standpoint. 

The  focus  during  this  step  of  the  prototype  development 
was  on  creating  a  KBS  which  would  provide  fast  and  user- 
friendly  access  to  the  plethora  of  equipment  guidance  found 
in  several  different  official  sources.  Attempts  were  made  to 
incorporate  expert  advice  (coming  from  SSgt  Jackson)  to 
supplement  the  regulatory  guidance  provided,  thus  providing  a 
source  of  information  more  like  that  found  when  consulting 
with  an  actual  expert  in  the  equipment  management  field. 

As  each  new  module  completed  the  initial  coding  phase, 
SSgt  Jackson  was  consulted  to  ensure  that  the  module  in 
question  was  providing  correct  answers  to  basic  user 
queries.  Syntactical  errors  identified  by  "VP-Expert"  were 
dealt  with  as  they  occurred. 

The  KBS  exhibited  ever-increasing  capabilities  as  new 
modules  were  programed;  the  module  programming  phase,  when 
finished,  led  to  the  determination  that  the  initial  prototype 
KBS  was  complete.  This  "first-draft"  KBS  was  then  given  the 
name  of  "The  Equipment  Management  System  Management  Advisory 
Program"  (or  EMS  MAP),  and  was  presented  to  SSgt  Jackson  for 
initial  testing  and  validation. 
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step  4: _ aM  Verification 

Type  of  Val idati on/ Verification.  In  order  to  validate 
and  verify  the  KBS  created,  the  researcher  selected  a  multi¬ 
level  process.  For  maximum  effectiveness,  a  single  equipment 
expert  at  the  base,  MAJCOM,  and  HQ  USAF  level  was  selected 
(based  on  the  recommendations  of  CMSgt  Conley,  HQ  USAF/LEYS, 
and  the  researcher's  base-level  equipment  experience).  This 
was  done  for  three  reasons: 

1.  To  ensure  that  the  prototype  KBS  achieved  its 
intended  cross-command  applicability. 

2.  To  enhance  the  liklihood  for  successful 
implementation  of  the  candidate  KBS  through  the  use  of 
experts  at  three  different  management  levels. 

3.  To  provide  additional  credibility  to  the  validation/ 
verification  process. 

Selection  of  Validating  Experts.  The  credentials  of  the 
experts  selected  and  the  rationale  behind  their  selection 
(beginning  at  the  HQ  USAF  level)  are  as  follows. 

To  act  as  the  USAF-level  validating  expert,  CMSgt 
Babbitt  was  selected.  Currently  serving  at  HQ  USAF/LGSS, 
CMSgt  Babbitt  combines  years  of  supply  experience  with  a 
relatively  powerful  position  within  the  overall  USAF  Supply 
community.  Serving  in  a  position  designed  to  oversee  the 
entire  Air  Force  Supply  System,  CMSgt  Babbitt's  opinion  on 
virtually  any  supply  matter  carries  considerable  weight. 
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SMSgt  Abeln,  serving  at  HQ  SAC/LGSEP,  was  selected  as 
the  MAJCOM-level  validation  expert.  His  selection  was  based 
on  several  factors: 

1.  HQ  SAC  was  selected  as  the  MAJCOM  to  be  used  for  the 
validation  process,  due  to  the  researcher's  experience  within 
this  command. 

2.  Key  managers  within  the  HQ  SAC  Supply  Directorate 
were  contacted  by  the  researcher  and  were  asked  to  recommend 
the  premier  equipment  expert  within  the  command's 
headquarters.  SMSgt  Abeln  was  the  overwhelming  choice,  based 
on  his  extensive  supply  experience  within  SAC  and  his  current 
participation  in  the  automated  Air  Force  Equipment  Management 
System  (AFEMS)  project. 

3.  SMSgt  Abeln  was  then  contacted  and  asked  to 
participate  in  the  validation  of  the  EMS  MAP;  his  willingness 
to  participate  in  the  project  finalised  his  selection  as  the 
MAJCOM-level  validation  expert. 

Finally,  CMSgt  Robert  Fatula,  serving  within  the  384th 
Supply  Squadron  at  McConnell  Air  Force  Base  (SAC),  was 
selected  as  the  base-level  validation  expert.  The  researcher 
selected  this  individual  based  on  first-hand  knowledge  of  the 
equipment  experience  possessed  by  CMSgt  Fatula,  coupled  with 
the  individual's  willingness  to  participate  as  the  equipment 
validation  expert. 

Ihs  Validation/Verification  Process.  Once  the  experts 
to  be  used  in  the  validation/verification  process  were 
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selected  and  initially  contacted,  the  process  was  set  to 
begin.  A  letter  was  drafted  and  forwarded  to  each  of  the 
three  experts  along  with  a  copy  of  the  prototype  KBS.  This 
letter  (presented  in  Appendix  B)  briefly  described  the 
intended  purpose  of  the  "EMS  MAP”  program,  gave  instructions 
for  its  use,  and  requested  that  the  experts  test  the  program 
as  they  saw  fit.  Once  the  experts  had  completed  their 
testing  of  the  program,  they  were  asked  to  form  an  opinion  of 
the  validity  of  the  program  based  on  the  following  general 
guidelines : 

1.  What  is  your  overall  impression  of  the  program? 

2.  Remembering  that  the  program  was  written  for  use  by 
novice  equipment  custodians,  do  you  think  the 
program  (as  it  stands)  would  be  useful? 

3.  What  items  appearing  in  the  program  are  incorrect  or 
inappropriate?  (in  other  words,  what  would  you 
change  about  it  to  make  it  better). 

4.  Finally,  what  items  would  you  like  to  see  added  to 
the  program  to  make  it  more  useful  to  an 
inexperienced  equipment  custodian? 

Telephone  interviews  were  conducted  to  receive  feedback 
from  the  equipment  validators  at  the  base  and  HQ  USAF  levels. 
The  researcher  was  afforded  the  opportunity  to  meet  face-to- 
face  with  SMSgt  Abeln,  the  HQ  SAC  equipment  expert  acting  as 
the  MAJCOM-level  validator.  The  specific  recommendations 
made  by  SMSgt  Abeln  during  the  KBS  validation  visit,  as  well 
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as  the  recommendations  made  by  the  other  two  equipment 
validators,  are  presented  in  Appendix  C. 

After  receiving  feedback  from  each  of  the  equipment 
validators,  the  "EMS  MAP"  was  updated  to  incorporate  the 
recommendations  given  by  the  experts.  After  these  changes 
had  been  made,  a  session  was  conducted  with  SSgt  Jackson  (the 
original  expert  used  to  create  the  program)  to  ensure  the 
changes  had  not  altered  the  basic  intent  of  the  program  from 
this  individual's  standpoint. 

As  a  final  step  in  the  validation/verification  process, 
the  completed  KBS  prototype  was  tested  using  a  non-random 
selection  of  newly  trained  equipment  custodians.  This  step 
was  intended  to  determine  whether  or  not  the  completed 
program,  developed  by  equipment  experts,  would  be  useable  by 
individuals  less-knowledgeable  in  equipment  procedures. 

To  begin  this  step,  volunteers  were  solicited  from 
a  group  of  individuals  attending  the  initial  equipment 
custodian  training  within  the  Training  Section  of  Wright- 
Patterson's  Base  Supply  account.  Five  individuals  agreed  to 
assist  in  the  validation  of  the  KBS,  which  took  place 
immediately  after  the  training  session  was  completed.  Having 
received  the  only  formal  equipment  training  available,  these 
individuals  appeared  to  fairly  represent  the  category  of 
novice  equipment  custodians. 

Each  of  the  novice  equipment  custodians  who  volunteered 
was  afforded  the  opportunity  to  work  with  the  prototype  KBS 
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for  approximately  15  minutes.  This  amount  of  time  was 
sufficient  for  the  volunteers  to  observe  the  overall  control 
structure  of  the  program,  and  to  receive  advice  from  the  KBS 
on  a  number  of  issues  pertinent  to  an  equipment  custodian's 
daily  tasks.  Through  interface  with  the  novice  equipment 
custodians  and  SSgt  Jackson  (the  original  equipment  expert), 
the  researcher  was  able  to  determine  that  the  prototype  KBS 
provided  adequate  user-friendliness  for  newly  trained 
equipment  custodians.  Further,  based  on  the  replies  from  the 
volunteers,  the  information  presented  by  the  program  appeared 
to  be  very  useful  in  performing  an  equipment  custodian's 
everyday  tasks.  Thus,  no  changes  co  the  program  were 
required  as  a  result  of  this  final  validation/verification 
step. 

With  the  completion  of  the  validation/verification  step 
of  prototype  development,  the  first-generation  "EMS  MAP" 
prototype  KBS  was  deemed  complete. 

Summary 

Chapter  IV  presented  a  more-specific  overview  of  the 
steps  taken  to  answer  the  research  questions  posed  in 
Chapter  I.  These  steps  were  comprised  of: 

Step  1:  KBS  Candidate  Selection 
Step  2:  Problem  Familiarization 
Step  3:  Prototype  Development 
Step  4:  Validation  and  Verification 
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specific  actions  taken  during  each  of  these  four  steps 
was  outlined  in  Chapter  IV.  The  results  of  these 
methodological  steps,  culminating  with  the  initial  prototype 
completion  of  the  "Equipment  Management  System  Management 
Advisory  Program,"  were  presented  in  Chapter  IV. 

Chapter  V  will  discuss  the  overall  conclusions  of  this 
study,  specifically  dealing  with  how  this  research  process 
has  answered  the  three  research  question  posed  in  Chapter  I. 
Finally,  recommendations  which  have  resulted  from  the 
research  conducted  in  this  thesis  will  be  presented. 
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V .  Conclusions  and  Recoironendat i ons 


Overview 

Chapter  V  begins  with  a  presentation  of  the  overall 
conclusions  of  the  research,  relating  these  conclusions  back 
to  the  original  research  questions  posed  in  Chapter  I. 
Additional  conclusions  made  by  the  researcher  are  also 
presented.  Chapter  V  also  presents  three  recommendations  for 
further  studies  related  to  the  field  of  artificial 
intel ligence. 

Conclusions  of  the  Research 

In  terms  of  the  three  research  questions  posed  in 
Chapter  I,  the  research  conducted  in  this  thesis  (as 
summarized  in  Chapters  1  through  IV)  has  yielded  the 
following  conclusions: 

Research  Question  1 .  Do  supply-related  tasks  lend 
themselves  to  KBS  development?  As  introduced  in  Chapter  I 
and  further  developed  in  Chapter  II,  the  Standard  Base  Supply 
System  represents  an  organization  which  could  indeed  benefit 
from  the  introduction  of  KBS  technology. 

Specifically,  it  was  found  that  organizations 
characterized  by  a  relative  shortage  of  experts  and  a  rigid, 
manual -driven  set  of  procedures  were  prime  candidates  for  KBS 
development.  Through  discussion  in  Chapters  I  and  II,  the 
SBSS  was  shown  as  just  such  an  organization.  Within  the 
Standard  Base  Supply  System,  the  training  of  equipment 
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custodians  was  identified  as  one  specific  task  which  could 
benefit  from  the  introduction  of  KBS  technology. 

Thus,  in  answer  to  the  first  research  question,  supply- 
related  tasks  do  lend  themselves  to  KBS  development. 

Research  Question  2 .  If  supply-related  tasks  lend 
themselves  to  KBSs,  how  can  a  prototype  KBS  be  developed 
within  the  Standard  Base  Supply  System?  As  illustrated  by 
Chapter  IV,  the  prototype  KBS  titled  the  "Equipment 
Management  System  Management  Advisory  Program  (EMS  MAP)"  was 
created  by  the  researcher.  Using  an  expert  from  Hright- 
Patterson  Air  Force  Base's  Base  Supply  Training  Section  and 
various  manuals  and  regulations,  the  EMS  MAP  provides  answers 
to  questions  commonly  posed  by  equipment  custodians  in  their 
day-to-day  account  management. 

Thus,  Research  Question  2  was  also  confirmed  with  the 
development  of  the  KBS  titled  EMS  MAP. 

Research  Question  3 .  How  can  this  prototype  KBS  be 
validated?  As  presented  in  Chapter  IV,  the  prototype  KBS 
developed  in  this  research  was  put  through  a  multi-level 
verification/validation  process.  Using  independent  equipment 
experts  at  the  HQ  USAF,  HQ  SAC,  and  base  levels,  the  validity 
of  the  prototype  KBS  was  confirmed,  culminating  in  the 
initial  completion  and  acceptance  of  thi  CMS  MAP  within 
Hright-Patterson's  Base  Supply  Training  Section. 

Thus,  Research  Question  3  was  also  answered 
affirmatively. 
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Additional  Conclusions .  With  the  resolution  of  the 
three  research  questions  posed  in  Chapter  I,  this  thesis  was 
nearly  complete.  However,  two  additional  conclusions  were 
drawn  by  the  researcher. 

First,  in  the  course  of  conducting  research  into  the 
topic  of  artificial  intelligence,  expert  systems,  and 
knowledge-based  systems,  the  researcher  found  that  these 
mysterious-sounding  computer  sub-fields  possess  a  gr^nt  deal 
of  pragmatic  and  relatively  straightforward  value  in  everyday 
computer  applications.  However,  the  concept  of  applying 
artificial  intelligence  techniques  within  the  Standard  Base 
Supply  System  was  considered  a  truly  revolutionary  concept  to 
many  supply  experts  consulted  by  the  researcher.  At  the  same 
time,  however,  these  same  experts  were  actively  involved  in 
implementing  numerous  small-scale  PC  applications  involving 
data-base  management  techniques. 

Thus,  it  appears  that  the  ''demystification"  of  the 
practical  uses  of  expert  system  technology  within  our 
logistics  areas  has  not  been  completely  successful  yet. 

Unless  the  mystery  of  this  fertile  subfield  of  computer 
technology  is  removed  from  the  minds  of  potential  users, 
numerous  candidate  processes  for  KBS  appli';ation  will 
continue  to  be  completed  in  a  less-efficient,  manual  manner. 

With  the  downsizing  of  our  military  forces  in  post- 
Operation  Desert  Storm  America,  every  effort  must  be  made  to 
capture  the  dwindling  expertise  still  possessed  and  to  press 
forward  with  all  potentially  lucrative  sources  of  increased 
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efficiency.  One  way  to  do  this  is  through  the  expanded  use 
of  expert  and  knowledge-based  system  technology. 

A  second  conclusion  was  drawn  by  the  researcher 
regarding  the  actual  prototype  KBS  development  process.  It 
was  determined  that  the  application  of  expert  system 
technology  within  the  logistics  doctrines  need  not  involve 
great  expense  in  either  actual  dollars  or  manpower. 

The  KBS  developed  in  this  thesis,  while  somewhat  limited 
in  scope,  demonstrates  the  enormous  potential  for  KBS 
technology  in  automating  many  small-scale  problems.  While 
extremely  large-scale  expert  systems  exist  which  have 
required  a  great  deal  of  time  md  money  to  develop,  it  was 
found  that  many  small-scale  problems  existed  within  the 
logistics  functions  which  could  be  solved  via  a  small-scale 
expert  system.  Thus,  a  continued  focus  should  be  placed  on 
expert  system  education  in  an  effort  to  provide  a  catalyst 
for  the  development  of  additional  small-scale  expert  systems. 

Recommended  Follow-On  Studies 

In  completing  this  thesis,  several  additional  avenues  of 
potentially  useful  research  became  evident  to  the  researcher. 
These  additional  research  projects  include: 

Expert  System  Roadmaps .  While  it  was  relatively  easy  to 
select  a  candidate  problem  for  KBS  implementation,  it  appears 
useful  for  research  to  be  conducted  to  develop  some  sort  of 
roadmap  for  future  expansion  of  expert  and  knowledge-based 
technology  within  each  of  the  logistics  areas  (Supply, 
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Transportation,  Maintenance,  and  Logistics  Plans).  Hinted  at 
by  Colonel  Blazer  in  his  AFIT  Form  53  ("External  Proposal  - 
Thesis  Research  Topic,"  see  Appendix  A),  a  need  exists  to 
"identify  potential  expert  systems  for  each  of  the  logistics 
functional  areas"  (Blazer,  1990).  This  research  could 
include  the  development  and  testing  of  a  standardized 
methodology  for  prioritizing  potential  logistics  KBS 
applications  into  a  top-down  list  of  good  candidates. 

Determining  the  Level  of  Understanding  of  Expert  System 
Concepts  Within  the  Logistics  Areas .  Research  in  this  area 
might  involve  a  survey  of  individuals  working  within  each  of 
the  logistics  disciplines  to  determine  the  level  of  knowledge 
and/or  fear  of  artificial  intelligence  techniques.  In  line 
with  comments  made  in  the  "Additional  Conclusions"  section  of 
this  chapter,  it  is  felt  by  the  researcher  that  ignorance 
prevails  within  many  logistics  organizations  as  to  the 
practical  value  and  relative  simplicity  of  many  knowledge- 
based  computer  programs.  The  true  state  of  knowledge 
regarding  expert  systems  technology  would  be  assessed  in  this 
recommended  research.  If  knowledge  of  expert  systems 
applications  was  found  to  be  lacking  within  logistics 
organizations,  methods  to  increase  the  knowledge  level  or 
reduce  the  fear  level  might  be  explored  in  this  follow-on 
research. 

Additional  Prototype  Expert  Systems .  The  logistics  area 
(and,  in  particular,  the  Standard  Base  Supply  System)  was 
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found  to  be  an  area  possessing  numerous  processes  which  could 
be  made  more  efficient  via  expert  or  knowledge-based  system 
application.  The  research  conducted  here  demonstrated  the 
feasibility  of  small-scale  prototype  development  of  KBSs 
using  only  limited  means.  Thus,  it  is  imperative  that 
additional  KBS  applications  be  completed  within  the  logistics 
arena  to  continue  to  streamline  our  logistics  processes  and 
stem  the  loss  of  expert  knowledge. 

Also,  as  noted  in  the  "Additional  Conclusions"  section 
of  this  chapter,  expert  or  knowledge-based  systems  can  be 
developed  to  solve  small-scale  logistics  problems  with  very 
limited  dollar  and  manpower  requirements.  Chapters  III  and 
IV  of  this  thesis  provide  a  soxind  methodology  for  developing 
and  validating  a  small-scale  KBS.  Thus,  additional  research 
is  needed  to  develop  and  validate  small-scale  expert  systems 
within  each  of  the  logistics  functions. 

Summary 

Chapter  V  began  by  presenting  the  conclusions  of  the 
research,  including  an  answer  to  each  of  the  Chapter  I 
Research  Questions.  Two  additional  conclusions  which 
resulted  from  the  research  process  were  also  discussed. 

Chapter  V  concluded  with  three  recommendations  for 
follow-on  studies  related  to  the  field  of  artificial 
intelligence  within  the  U.S.  military  logistics  structure. 
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ADoendix  A:  AFIT  Form  53 


FOR  LSB  use  only 


EXTERNAL  PROPOSAL  •  THESIS  RESEARCH  TOPIC 

School  uf  Syiiffni  and  LecUtkt 
Ah  Fofr<  Iniihitic  of  Technolo|y  (AU) 

Wfiy'hi-Patictinn  ATB  OH  4543J 


O^tC  AMCA 

Artificial 

Intelligence 


(^0-9^ 


Col  D.  Blazer 


HO  OSAF/LEXX 


Ay WgiW# 


Expert  Systems  in  Logistics  Processes 


a.  Identify  potential  expert  systems  for  each  of  the  logistics 
functional  areas. 

b.  And, or  Build  a  prototype  expert  system  for  a  particular  logistics 

process  application.  i 


t.  neCAte  iNoie  Ate  tmc  nAtie  r acton.  NCLATtoNtMine.  amo  tiruAtioNt  tMvetveo.  rt't#  •  aAttf  •/  pmptr.  f/ 


Expert  Systems  are  computer  programs  that  guide  a  novice  through  a  series 
of  decisions  so  the  novice  can  make  expert  decisions.  There  are  an  endless 
number  of  potential  applications  for  expert  systems  in  all  the  logistics 
functional  area  processes.  Research  in  this  area  could  include  a  review  of 
a  logistics  function  (supply,  transportation,  maintenance,  engineering, 
logplans,  or  contracting)  to  identify  potential  expert  system  applications. 
The  report  would  provide  a  roadmap  for  development  of  expert  systems. 
Other  research  efforts  would  include  the  development  of  expert  systems  for 
specific  applications  (for  example  the  contractor  buying  or  the  supply 
stock  fund  manager). 


•  TkCAtt  IMOICATC  IHOIVIOUAUf.  fTUOIft.  ON  OTHfN  AU  •  W'C  A  Tie*s  f  THAT  CAN  TUNNIfM  NCtOUNCCI  TON  THIS  NCtCANCM 

Depends  on  the  functional  area  selected,  but  other  sources  include:  AFLC 
HQ/XP  and  MMI,  AFLMC,  HQ  USAF/LEYS 
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Appendix  Ij.  Sample  Expert  Validator  Letter 


From:  Capt  Rick  Nelson  13  Jul  91 

Subject:  Computer  Program  Validation 
To:  Equipment  Expert  Validators 


1.  Here's  the  computer  program  we  talked  about  today.  As  I 
mentioned,  this  program  is  a  part  of  my  Master's  Thesis  here 
at  AFIT,  and  your  help  in  evaluating/validating/improving  it 
will  help  me  out  quite  a  bit.  The  program  itself  should  be 
fairly  self-explanatory.  To  rtin  the  program,  all  you  need  to 
do  iw  to  insert  the  diskette  into  your  computer's  A  drive, 
type  in  "Run"  and  press  Return. 

2.  How  you  can  help  me  is  by  going  through  the  program  for  a 
while,  looking  at  its  capabilities  and  the  actual  information 
I've  coded  into  the  program  to  make  sure  everything  makes 
sense  from  an  equipment  standpoint.  If  you  see  things  in  the 
program  which  are  incorrect  (or  just  simply  inappropriate,  in 
your  opinion),  make  a  note  of  the  specific  item  in  order  to 
answer  the  questions,  below. 

3.  When  you  get  done  checking  out  the  program  to  your 
satisfaction,  please  be  prepared  to  discuss  the  program  with 
me  using  the  following  questions  for  guidelines: 

a.  What  is  your  overall  impression  of  the  program? 

b.  Remembering  that  the  program  was  written  for  use  by 
novice  equipment  custodians,  do  you  think  the 
program  (as  it  stands)  would  be  useful? 

c.  What  items  appearing  in  the  program  are  incorrect  or 
inappropriate?  (in  other  words,  what  would  you 
change  about  it  to  make  it  better). 

d.  Finally,  what  items  would  you  like  to  see  added  to 
the  program  to  make  it  more  useful  to  an 
inexperienced  equipment  custodian? 

4.  If  you  have  any  questions,  you  can  get  a  message  through 
to  me  by  calling  AV  785-8989. 


RICHARD  G.  NELSON,  Capt,  USAF  Atch: 

EMS  MAP  Program 
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